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ABSTRACT 

Technical problems involved in obtaining scores for 
essays written by teacher candidates are addressed. The rating 
procedure assures that each essay is read by three raters specially 
trained for the task. Ratings range from "1 " for _incompetent or 
unacceptable to 11 <J" f qroutstanding. The score assigned to each essay 
willbe the sum of the ratings assigned it_by the three raters and 
may range from "3" to "12^" If needed^ a fourth reader's rating shall 
repiace_the most _discrepant of the three raters* ratings^ The 
following three components comprise a uniform system for keeping 
track of the iovement of several thousand essays to be rated through 
the hands of perhaps several dozen raters: (1) a detailed chronology 
of the steps that must be taken to prepare the essays, assign them to 
ratings teams , and move them between raters within teams: (21 copies 
of the various forms that will be used in the ratings process, 
completely illustrating the processes step by step; and (3) blank 
copies of the forms to be duplicated for use as prescribed. (RL) 
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GLOSSARY 



To avoid confusion, the terms specified below are 
used consistently in accordance with the acocetpanying definitions. 

FATING: this term will be used to refer to the 
numeric sign that a rater awards an essay to indicate 
his or her judgment of its quality; a rating may range 
frcm "1" for unsatisfactory to "4" for outstarrling. 

SCORE: this terai will be used to refer to the sura of 
the ratings awarded to an essay by the three raters who 
have read it; scores may range frdn "3" to "12". 

GRADE: this term will be used to refer to the interpretation 
of a score as "satisfactory" (S) or •"unsatisfactory" (U) ; 
it is reoonroended that the U designation apply to scores of 
"3" and "4", with the S designation applying to scares fran 
"5" through "12". 
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1. OVERVIEW. This manual is addressed to the technical problems that 
will be involved in obtaining scores for the essays written by candidates 
for this subtest. The rating process is straightforward, but it is 
complicated in its execution, simply because there will be a large 
number of people involved and a great many forms and essays to be handled. 

The rating procedure is this. Each essay will be read by 
three raters who will be specially trained for the task. The ratings 
will range from "1" for incompetent or unacceptable up to "4" for out- 
standing. The score assigned to each essay will be the sum of the 
ratings assigned it by the three raters, so scores on any essay may 
range from 3 to 12. (The sum is used rather than the average score 
because it is easier to compute and simpler to use in various calculations and 
in reporting scores .) In cases in which the ratings assigned to an 
essay by the three raters diverge beyond certain specified limits, the 
essay will be read by a fourth reader, henceforth to be called the 
"referee," whose rating shall replace the most discrepant of the three 

raters* ratings. 

Since there will, on any particular occasion, be several 
thousand essays to be rated and perhaps several dozen raters, and since 
it is essential that each rater be aware neither of the identity of the 
writer whose essay he or she is reading nor of the ratings assigned the 
essay by other raters* the heed for a uniform system for keeping track 
of the movement of essays through the hands of the various raters is 
obvious . 

The system that is outlined in the following pages has, 
essentially, three components. 
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2. Stamp or write the code number of 
the Competency Examination with which 
the test booklet is to be associated 
in the upper right-hand corner of the 
cover sheet AND of each attached blank 
sheet. 



B. Exposition. The necessary sections of explanatory 
prose will be inserted between the segments of the 
chronology at the points where they are logically called 
for. Each such section will be identified by a number 
and a descriptive heading in capital letters. 

C. Figures. These include specimens of forms and materials 
to be used in the ratings task; forms partially completed 

to illustrate, step-by-step, activities described in the 
Chronology; and various tables and checklists that will 
simplify particular tasks. These figures will be numbered 
in sequence as they are referred to in the text, and all 
figures will be found, in numerical order, in Appendix A 
to this booklet. A blank copy of each separate form that 
will have to be duplicated will also be found in Appendix 
B, so that forms may be removed for duplication without 
disturbing the sequence in Appendix A. 

References to figures will be made as frequently as 
necessary in the following manners. 
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A. A detailed CHRONOLOGY of the steps that must be taken 
by the staff managing the ratings process to prepare the 
essays, assign them to ratings teams * move them between 
raters within teams, and so forth. 

B. Blocks of EXPOSITION— small essays like the present 
one—that provide essential information or explain how 
and why a certain operation is to be carried out. 

C. Various FIGURES — forms, charts, and illustrations — to 
assist in understanding the points made in the other 
sections. 



2. FORMAT. For the sake of easy reference, these three sorts of materials 

will be distinguished typographically . 

A. Chronology. The steps in completing a particular 
sub task in the ratings process will be numbered, set up 
in a ruled box, and identified by a descriptive heading . 
For example, the steps in the preparation of the writing 
subtest instrument would be presented in this manner: 



Preparing the Test Bookle t 

1. Staple three sheets of lined 3-1/2" 

x 11" writing paper to each cover sheet. 
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1. Within the chronology, the references to figures will 
be inserted in a ruled arrow pointing toward the numbered 
step to which the figure is pertinent. For example: 





Preparing the Test Booklet 




See FIGURE 1 


1. Staple three sheets of 


lined 




8-1/2" x 11" writing paper 


to 




each cover sheet. 











2. Within the Exposition, references to figures will be 
inserted within the text where called for. For example: 

"The code number for a set of papers are to be written 
consecutively down column 5. SEE FIGURE 4." 



3. PREPARATION OF THE ESSAYS FOR THE RATERS. Each essay will have a 
cover sheet stapled to it. This cover sheet will contain biographical 
data about the writer of the essay, the test instructions, a list of 
possible essay topics, and—most importantly for tracking purposes—a 
code number. (This code number is common to all the subtests taken by 
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the candidate, and the greatest care must be taken that the score awarded 
the candidate's essay is accurately associated with his proper code 
number . ) 

The candidates 1 essays will be assembled into sets of about thirty, the 
cover sheets will be removed and filed — so that the essays will be 
graded anonymously — and the candidates' code numbers will be recorded on 
a Rater's Tally Sheet, one of which will be assigned to each set of 
thirty or so essays. 

The Rater's Tally Sheet (FIGURE 1) is a multipurpose form that will be 
used for tallying individual rater's ratings, recording referee's ratings, 
and computing total scores. The data on the Summary copy of the Rater's 
Tally Sheet will also be used for making computations of rater agreement 
and by the keypunch operator who prepares the candidates' scores for the 
computer. Turn now to the sample Rater's Tally Sheet, Figure 1 in 
Appendix A, and study it before proceeding The form will hereafter 
usually be referred to by the initials RTS. 

For convenience's sake, it will be asa*i:ii-».d in the following discussions 
that there are 1*000 essays to be rated- 



Assembling the Sets of Essays 



1. Separate the 1*000 essays 
into sets of thirty. This will 
yield 33 sets with ten essays 
left over. Distribute the ten 
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See FIGURE 1 



See 


FIGURE 2, 


where 


this_ 


has 


been done 


for a 


sample 


set 









remaining essays, one essay per 
set, among the first ten sets. 

2. Place a blank RTS atop 
each set of essays. 

3. Write the number of the 

set prominently In the upper right 
hand corner of the RTS. 

Recording the Code Numbers 

4. For each set of essays, copy, 
from the cover sheets of the test 
booklets , the candidates 1 code 
numbers and list them consecutively 
down Column 5 of the RTS. 
When this is completed, each of 
the 33 RTL f s will contain the 
the code numbers of the essays 
in the set in the same order 
that the essays themselves 
are arranged . 

Filing the Cover Sheets 

5. Remove the cover sheets from 
the writing test booklets, keep- 
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ing them together by sets. 
NOTE: While removing the cover 
sheets, check to see that the 
code number appears on each page 
of the essay. If a number is 
missing, write it in. 

6. Place a blank sheet of paper 
atop each set of cover sheets and 
write the set number prominently 
on it. 

7. Fasten the cover sheets together 
by sets, and file them consecutively 
by number. 



A total of five RTS's will be needed for each set of essays: one for 
use by each of the three raters, one for use by the referee, and one for 
compiling scores. (Before the sheets are duplicated, however, the sets 
of essays will be assigned to teams of raters, so that the testees 1 ;ode 
numbers will have to be copied on the RTS only once.) 

The instructions for preparing the essays must, therefore, be interrupted 
at this point for a discussion of, first, a method of estimating the 
required number of teais of raters, and, second, a system for assigning 
the sets of essays to teams of raters. The instructions resume with 
Section 7. 
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4. ESTIMATING REQUIRED NUMBER OF TEAMS OF RATERS. When the number of 
essays to be rated is known, it is possible to determine how many teams 
of raters will be required for how many days. 

Experience has shown that it will take an experienced rater approximately 
two-and-one-half to three minutes to read the 300 to 500 word essay the 
average candidate will write during the examination period. We are 
assuming 1,000 essays, each of which is to be read by three raters in 
three minutes. The rating process, then, will take approximately 9,000 
man-minutes or 150 man-hours. 

These 150 man-hours, however , cannot be broken down into solid 
eight-hour workdays. Since reader fatigue is a major threat to the 
reliability of ratings, rest periods must be provided. The highest 
efficiency can probably be obtained by scheduling ratings sessions of 
ninety minutes, during which thirty papers will be read. 

A minimum of four and a max5.mum of five such ratings sessions can 
be scheduled in a working day, depending on the length of breaks. Since 
time will often be required for discussions of ratings problems and for 
retraining, it is best to figure on scheduling four ninety minute ratings 
sessions per day. 

The effective working day, then, is 4 x 90 minutes - 360 minutes = 
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six hours. The required 150 man-hours translates into 25 six-hour man- 
days, which must be further divided by three to obtain the number of 
teanFdays required. This will be eight-and-one- third. 
To summarize: 

1*000 essays x 3 readers x 3 minutes 1,000 essays 

= = 8.33 team-days. 

60 minutes x 6 hours x 3 team members 120 essay s-per^ day 

Or, in general: 
Number of Essays 

— = Number of Team-Days. 

120 

The next question is how to balance off the number of teams and the 
number of days. For example* eight teams (24 persons) could rate 1,000 
essays in one extended work day. But 24 is probably too large a group 
to train conveniently and too large to house and provide workspace for. 
Alternatively* one three-person team could grade 1,000 essays in about 
eight days* but that would be wasteful in terms of the administrative 
and support personnel that would have to be provided. The best solution 
would be three teams, which could rate the 1,000 essays in a bit less 
than three days. (8.33 team-day s/ 3 teams =2.78 days.) 

A ^ood rule of thumb , then, is to calculate that for every 4^0a 
essays the services of nine raters will be needed for three days . (And 
for each nine raters , as explained below, there will be three referees.) 
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5. ASSIGNING SETS OF ESSAYS TO TEAMS OF RATERS, This is not a complicated 
process, but it must be handled carefully. 

A blank Essay Management Schedule (hereafter, EMS) is presented as 
Figure 3 in Appendix A. (FIGURE 3) . It will allow for scheduling the 
ratings of 1,000 or so essays. Each of the required nine raters will be 
assigned to a team (I, 2, or 3) and each will be assigned a letter (A, 
B, or e) within that team. Each rater f s name will be written in the 
appropriate space on the EMS. 

REFER NOW TO FIGURE 4. In FIGURE 4, sets of essays have been 
assigned, by numbers, to members of the rating teams for consecutive 
readings. Note that a "blocks of nine" pattern has been followed for 
assigning three sets of papers to three raters in a team. By following 
this pattern it is simple to assign and distribute the essays and keep 
track of their movement during the ratings process. 

For each block of three ratings sessions, three consecutively 
numbered sets of essays are assigned to each team of raters. In the 
example in Figure 4, for instance, the sets of essays numbered 1 through 
3 are assigned to team 1 for the first three ratings sessions. The 
lowest numbered set of the three is given to rater A, the second to 
rater B, the highest to rater C. The essays are collected at the end of 
the first ratings session and redistributed prior to the beginning of 
the second session. 
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Fbr the second ratings session* rater A's set of essays goes to 
rater C, C f s set to B, and B's set to A. The same pattern of movement 
is followed in distibuting sets of essays to the raters prior to the 
third ratings session: A f s to C, C ? s to B, B f s to A; as follows: 



Ratings Session No. 

1 2- 



A 1 ^ 2 ^ 3 

Rater B 2- ; " X ^ 3 V 1 

C 3. *±. ^2 



Or, for a later session: 



Ratings Session No. 

| 2 3 

A 19. ^ 20. ^ 21 

Rater B 20 - 21 ^19 

- " * T_ \ ^ 

G 21-^ ' 19 20 



The "blocks of nine 11 pattern, then, is to be followed both in 

initially assigning sets of essays to raters and later y during the 

ratings sessions, in keeping track of the movement of sets of essays 
among teams of raters. 
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6. ADDITIONAL OBSERVATIONS ON THE ESSAY MANAGEMENT SCHEDULE. Note in 
FIGURE 4 that the schedule makes provision for an initial training 
session of about two hours and for one or possibly two retraining sessions 
if they are needed. Note also that the completed schedule calls for 
five ratings sessions on the final day, so that the work may be completed 
without extending into a fourth day. If the retraining session for 
which time is provided on the second day is not required, the extra 
ratings session could be scheduled on the second day, which would probably 
be to the liking of raters having to make travel connections. 

Note also that the raters in Team 3 will have completed their work 
(in this case) by mid-day oh the third day and could be released. 
Raters with travel connections to make or with other reasons for leaving 
early should be assigned to Team 3. 

The Essay Management Schedule is completed some time prior to the 
time when the raters will be assembled; it lays out the details of the 
ratings task in such a way that, for instance, assignment of raters to 
teams may be done in ways that take cognizance of the needs both of 
raters and administrative staff. 
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7. FURTHER PREPARATION OF THE SETS OF ESSAYS. Once the EMS's have been 
completed, the Information on them will be used to complete the preparation 
of the Rater's Tally Sheets for the raters. 













Assigning Sets of Essays to Raters 




L Assign each rater to a team, 




1, 2, or 3. 




2. Assign a letter — A, B, or C-- -to 




each rater on each team. 




3. Insert the raters' names in the 




proper spaces on the EMS . 


1 See FIGURE 4 where this 


4. Following the procedure discussed 


1 has been done.- 






in Section 5 above and illustrated in 




Figure A in Appendix A, assign sets of 




essays to raters within teams. 



Continuing Preparation of t he Rateg^g. 
Tally Sheets 



1. Use the information on the EMS to complete 
this step. In the space provided at the top 
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See FT GUILE 5; wher e these 
two sttf.ps have been completed 



of column 2 of each RTS write in the 
number of the team to which a set 
of essays has been assigned, 

2. In the spaces provided in column 2 
of each RTS, write the names of the 

three raters oh a team behind the 
letter to which each has been assigned. 
NOTE: If names are not available at 
this stage, ignore step 2 and write in 
the names after the teams are identified 
or, even, just before distributing the 
sets of essays to the raters. 

3. For each set of essays, prepare a 
folder by writing prominently on 

its cover the Set Number. of the set of 
essays it will contain. 

4. Xerox four copies of each RTS 
thus prepared. 

5. Circle in red on one copy reader 
A's name. 

6. Circle in red on a second copy 
reader B T s name. 

7. Circle in read on a third copy reader 
C's name. 

8. Circle in red the column heading 
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"Referee's Name" (Column 4) on a fourth 
copy. 

9. Write in large letters the word 
"Summary 11 at the top of the fifth copy. 

10. Place each set of essays with its 
accompanying five copies of the pre- 
pared RTS in its proper folder. 



The essays are now ready to be distributed to the raters. Before 
discussing the management of the ratings session, however, some brief 
comments about the physical arrangements are in order. 



8. PHYSICAL ARRANGEMENTS FOR THE RATINGS TASK. Workspace both for the 
raters and for the clerk or secretary who will be in charge of distributing 
the essays and compiling the scores must be considered. 

A. The conditions under which the raters read the 
essays should be, as much as possible, uniform both from one essay to 
another and from one rater to another. It is not satisfactory, that is 
to say, for the essays to be given to the raters to be read at the 
raters' homes or workplaces. Rather, the raters should be brought to a 
central location where uniformity of rating conditions can be controlled 
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and where random interruptions and distractions can be minimized. 

Ratings tasks of the present sort have commonly been 
carried out under one of two physical arrangements. In the first alternative, 
the raters pick up the items to be rated at some central location and 
then take them to some private place--typically a hotel room — for the 
reading. In the other alternative, all the raters are assigned workspace 
in some large room or hall and do their reading in this common area. 

Either alternative is satisf actory, since either provides 
for a general uniformity of conditions, but the latter arrangement seems 
likely to be usually the most convenient. 

B. The clerks and/or secretaries charged with handling 
the essays and making initial tabulations or scores should have a work 
station in or near the location where the raters do their reading. They 
will need table space adequate for laying out the folders containing the 
essays and for spreading out the contents of the folders to do the 
necessary tabulating. 

9. ADMINISTERING THE RATINGS SESSIONS. It seems likely that the 
person who is responsible for training the raters will also be in overall 
charge of the ratings task itself. Those portions of that person's 
duties that pertain to the training and retraining functions are treated 
in this manual. 

After the completion of the first ratings session, a good 
part of the work of compiling and recording the scores can be done during the 
ratings sessions, while the raters are at work, so that at the end of 
the final ratings session only the work done during that session will 
remain to be processed. The tasks for which the administrative staff 
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wiil be responsible during tie ratings sessions may conveniently be 
distinguished according to the points In the ratings process when they 
can or must be accomplished. 

A. Prior to each ratings session, essays must be distri- 
buted * and after each session they must be collected, and 
certain record-keeping operations must be performed. 

B. During each ratings session except the first, the 
ratings awarded by the raters during the preceding session 
must be transferred from the raters' copies of the Rater's 
Tally Sheet to the Summary copies of the RT3. (i.e. > the 
ratings given during the first ratings session will be 
compiled during the second session, and so on.) 

C. Following the conclusion of each three session block— 
i.e., each time the reading of a set of essays by all 
three raters has been completed — the ratings awarded to 
each set of essays must be inspected for three purposes: 

1. to determine which essays must be submitted to 
a referee; 

2. to compute and record the scores of the essays 
that will hot be submitted to a referee; and 

3. to make initial estimates of inter-rater agree- 
ment, for the purpose of determining whether rater 
retraining session should be scheduled and for which 
teams. 

Each of these tasks is discussed in turn in the following sections. 

21 
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10. ESSENTIAL OPERATIONS PRIOR TO AND FOLLOWING EACH RATINGS SESSION. 
Before each session* each set of essays will be removed from Its folder 
and will have clipped to it an RTS marked for the rater whose turn It is 
to read the essays. The sets of essays will be passed out to or picked 
up by the raters. Track will be kept of the movement of the essays on 
the Essay Management Schedule. 

After each session, the essays will be turned in by or 
collected from each rater, returned to the proper folder, and checked In 
on the Rater's Tally Sheet. If another rater is still to read the set 
of essays, the RTS for the next rater will be clipped to the set of 
essays, so the set is ready to be passed out at the start of the next 
session. 

These operations are straightforward, and their Chronology 

follows. 







Preparing a Set of Essavs-for a 






Rater 






1. Assume rater A of Team 1, who is to 


prefer to FIGURE 4 




read the set of essays marked 01. 






2. Remove the set of essays from the 






folder marked 01, along with rater A's 
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RTS , leaving trie ocner txio s in tne iuiull • 




3i Clip A 8 Klo to trie set or essays aicer 




/tka/tj/'(n/> f-ViCk TllimHorQ OT*l fVlP flVlPPf" 3.<?3.1_IlSt 

CnECivUTE tile CUQc llLllllU CL o vjii tiic oncci. »-*&»-* *- 




tnose on trie er>says in cne set) uu cubui c 




tnar a set or essays may not. nave dccu im» 




filed. 




4. When rater A picks up or is given cne 




ackt- n-F oaa ova Hrflu a sttiaI 1 circle around the 




number or the essay in trie column corresponding 




to the ratings session ana in tne row 






| Refer to FIGDRE_A 


roiiowing tne rater t» name un lug 1^110 • 




5. For each rater on each team repeat the 




preceding steps prior to each ratings 




session. 



Receiving a Set- of Essays from a Rater 
After a Ratings Session 

1. Assume rater A of Team 1, who has 
completed reading the set of essays marked 0i. 

2. When the set of essays is returned 

by or picked up from the rater, cross out the 
circle drawn around the number of the set on 
the EMS in step 4 above. 
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3. Remove the RTS from the set of essays 
and return it to the proper folder. 

4. If another rater is still to read 
the set of essays, clip his RTS to the 
set of essays and replace the set in 
the folder; if there is no other rater, 
return the set of essays to the folder and 
draw a circle around the number on the outside 
of the folder to indicate all initial ratings 
have been completed. 

5. For each rater on each team repeat 
the preceding steps at the close of each 
ratings session. 

6. A second class of operations to be 
carried out during the ratings session pertains 
to the total scores awarded to essays by 
three raters. This data, of course, will be 
available only at the end of a block of 

three ratings sessions. 

The operations in question are: 

a. the determination of which essays 
will have to be forwarded to a referee; 
and 

b. the totalling of scores of essays 
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that will not have to be forwarded to a 
referee. 



A discussion of the nature and timing of the referees' work is 
deferred until Section 13, below. 
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11. COMPILING SCORES AWARDED BY THE RATERS. Each time a set of scored 
essays is returned by a rater, the rater's ratings are to be copied from 
his or her Rater's Tally Sheet onto the Summary copy of the RTS. This 
should be done as soon as possible during the ratings. session following 
the one in which the ratings were awarded. Care must be taken to transcribe 
the ratings accurately from one RTS to the other; the numbers in the 
margins of the RTS (columns 1 and 8) are intended as an aid to accuracy. 



Compil ing Rater s* Scores on the 



See FIGURE 5, where this has 
been done for a fictitious 
set of scores _ 



Summary Rater's Ta lly Sheet 



1. Assume that Rater A of Team 1 

has read and returned the set of essays 
marked 01. 

2. Take Rater A f s RTS and the RTS 
marked "Summary" from the folder con- 
taining essay set 01. Carefully 
copy Rater A's ratings in column 2 
under Rater A's name. 

3. Assume that Raters B and C of 

Team <^ jftave completed their readings of 
t he set of essays matked 01, 
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4. Carefully copy Rater B's and Rater 


See FIGURE 6* 
been done for 
tious sets 


where this has 
three ficti- 
of scores. 


C T s ratings under their names in column 
2 of the "Summary" RTS. 






5. Always return all copies of the 






RTS promptly to the proper folder when 






the scores have been copied. 






6. Repeat these steps for each rater 






and each set of essays during the ratings 






session following the one in which the 






ratings are awarded. 



12. DETERMINING WHICH ESSAYS ARE TO BE READ BY A REFEREE. The referees 
are particularly expert and experienced judges of essays. In cases 
where the three raters of an essay do not agree within specified limits 
(as explained below), the essay is sent to a referee, whose score will 
replace the most discrepant of the three original scores. 

When all three raters in a team have read a set of essays and 
the ratings have been transcribed on the Summary copy of the Rater's 
Tally Sheet, each set of ratings must be examined to determine whether 
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the ratings fall within acceptable limits. If the three ratings are 
within these limits, they are summed and the total is entered in column 
3, which is headed: "Total or to Referee." If the thirea ratings diverge 
from one another by too much, an X is entered in column three on the 
Sugary Rater's Tally Sheet. When all the ratings on the Summary RTS 
have been so treated, the X f s are copied in the proper rows in column 3 
of the referee's copy of the Rater's Tally Sheet, as directions to the 
referee which essays in the set he is to read. 

The nature and the management of the referees' function are 
treated in detail in Section 13. The concern here will be with the 
method for deciding which essays are to be marked for reading by a 
referee. Two rules govern the decision whether of not to send an essay 
to a referee. 

A. If any rating in the set of three differs from another by 
two or more, the essay is to be sent to a referee. 

B. If one rater gives an essay a passing rating (2 or higher) 
while the other two give it a 11 V it is to be sent to a referee. That 
±3 to say, that all sets of ratings with distributions of 1 1 2 are to 
be read by a referee. 

The possible combinations of ratings for three raters is as 

follows: 

111 1 3 3 2 3 4 
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2 4 4 

3 3 3 
3 3 4 

3 4 4 

4 4 4 



Under Rule A, which says that if there is a spread of two 
points or more between any two scores, the essay shall be sent to a 
referee, those combinations that are underlined above must be read by a 
referee and would be marked with an X in column 3. In addition, under 
Rule B, the combinations 112, which involve disagreement about whether 
an essay passes or fails, would also be marked for the referee's attention 

(Note that the Order of the scores is irrelevant. For decision 
purposes, for example, the following sets of scores — reading across the 
columns headed by the raters' names-would be exactly equivalent: 2 3 4, 
3 4 2, and 4 3 2.) 




114 
12 2 
12 3 
12 4 



13 4 

JL4 4 

2 2 2 

2 2 3 

2 2 4 

2 3 3 



Several of these combinations are unlikely to occur — for 
instance, combinations such as 1 1 4 or 1 4 4 — and are included only for 
the sake of completeness. 

As many as one-third of all ratings awarded by the three 
original raters may be found to fall in combinations covered by one of 
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the rules above* and could therefore be marked for an additional reading 
by a referee. 



This step of evaluating-^mbinations of ratings to decide 
-and^which should be sent to referees is 
-importance; It is essential that the greatest 



care be taken in making the decisions » in totalling scores and in 
marking essays to be ^read by referees . Accuracy must be preferred 
to speed at this step, even if that means that numbers of the scores 
viil be processed^af ter rather than during the ratings sessions « 



Deciding oh Score Totals or Referrals 
to a Referee 



1. Be sure you know the rules governing 
the forwarding of essays to referees. 

2. Always use a ruler or straightedge 
when examining sets of ratings; make 
sure both ends of the ruler are lined 

up with the same numbers in the marginal 
columns . 

3. Examine each set of scores on 
the Summary RTS to see if it contains 
either two scores that differ from one 
another by two or more (Rule A) 
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See Figure 7 , where this has 
been done for a fictitious 
set of data; make sure 
you understand why each 
decision has been made. 



or is some combination of the ratings 
112 (Rule B). 

4. If a set of ratings is of one of 
these descriptions ^ mark an X on 

its row in column 3 of the Summary RTS. 

5. If a set of ratings does riot fit 
either of the descriptions, add them up 
and write the total score on 

its row in column 3 of the Summary 
RTS. 

6. Cdritiriue until all the sets of 
ratings on the Sheet have been treated. 

7. Take the Referee 1 s RTS from the 
folder, fold it lengthwise down the 
lefthand line defining column 3. 
Lay the Referee 1 s RTS on the Summary 
RTS so that the spaces in column 

3 match. 

8. Carefully mark an X in column 3 
of the Referee's RTS in every space 
corresponding to one on the Summary 
sheet that contains an X. 

9. Check all of your work. 

10. Clip the Referee f s RTS to the set 
of essays. 

11. Return everything to the folder. 
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13. THE REFEREES AND THEIR ROLE. The referees will be readers of broad 
experience and acknowledged professional expertise In the teaching of 
written composition. Most of them will be drawn from among senior 
members of high school and college English departments. Their role in 
the overall ratings process is to adjudicate disagreements among raters. 

In cases where raters have disagreed whether an essay should 
be passed or failed, the referee reads the essay for the sake of fairness 
to the testee. In cases where the ratings awarded by raters diverge 
greatly, the referee reads the essay for the purpose ~f reducing the 
discrepancy-the referee's score will replace the most discrepant original 
score. (Reducing such discrepancies will increase the reliability of 
the ratings process, which is to say, reduce the possibility of unfair 
or inaccurate scores being awarded to testees.) 

Several remarks about the referees are in order here. 

A. Since, as noted above, it would not be unusual for as many 
as brie-third of all the essays to be in need of a referee's adjudication, 
it would be good to plan on having one referee for every three teams of 
raters. It will then take the referees approximately the same amount of 
time to complete their readings as it will take the raters to complete 
theirs. 

B. Although it would be possible for the referees to do their 
reading at another time and place, there are three reasons for preferring 
that their work— or at least the bulk of it— be done during the regular 
ratings sessions • 
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1. The referees must go through the same training and 
retraining sessions as the raters, and may as well stay 
and get to work. 

2. If the referees are scheduled to be present during 
the ratings sessions, they will always be available to 
replace raters who, for whatever reasons, do not show up 
for ratings sessions. 

3. The consideration that readings of the essays should 
be made under conditions that are as uniform as possible 
applies to the referees as well as to the raters. 

Basically the same procedures that were followed earlier for 
assigning sets of essays to raters will be followed in assigning them to 
referees. The same procedures that were followed in passing out and 
collecting essays from raters will also be followed with the referees. 



Assigning Sets of Essays to Referees 

NOTE: Do not separate the essays to 
be read by a referee from the others 
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See FIGURE 8 for format 



in the set; the essays are to be kept 
together and in the original order, 
as this will expedite reassociating 
essays with their cover sheets. 
The raters and referees will have 
been given instructions to the same 
effect . 

i; Assume three teams of raters and 
three referees. 

2. Assign one referee to each team. 

3. Prepare a copy of the Referee's 
Routing Sheet for each referee, writing 
the referee f s name at the top and the 
numbers of the sets of essays assigned 
to the referee's team down column i. 

4. As a set of essays is issued to a 
referee: 

a. Write the referee f s name at 
the top of column 4 of the Rater f s 
Tally Sheet marked for the referee; 

b. Make a check mark in the "Out" 
column (column 3) on the Referee f s 
Routing Sheet atid fill in the date 
(column 5) in the row spaces 
corresponding to the number of the 
set of essays. 
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NOTE: The referee is to be given drily 
the set of essays and his or her copy 
of the RTS, so that rio referee will 
have information about how a rater 
may have scored any particular essay. 
All completed Rater's Tally Sheets 
should be retained in the folder* 

5. As a set of essays is received from 
a referee: 

a. Check off the set in column 
four of the Referee's Routing 
Sheet . 

b. Return the set of essays and 
the referee's Rater's Tally Sheet 
to the folder. 

6. Issue the referee his next set of 
essays, repeating the above steps until 
all sets of essays have been read by the 
referees • 



15. RECORDING THE REFEREE'S RATING . When a referee has read and rated 
the essays in a set and recorded the ratings on his or her copy of a 
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Rater's Tally Sheet, the ratings are to 



be transferred to the Summary 



copy of the Rater f s Tally Sheet for the 



particular set of essays. 



Recording a Referee's Rating 

1. Remove the referee's RTS 
from the set of essays • 

2. Replace the essays in the folder, 
and take the Summary copy of the 

RTS from the folder. 

3. Fold the referee's copy of 
the RTS down the right hand margin 
of Column 4 (as was done earlier), 



his ratings in the proper spaces. 
4. When the ratings are copied, re 
place the referee's copy of the RTS 
in the folder. 



See FIGURE 9 where this 
has been completed. 



and carefully copy 



15. REPLACING THE DISCREPANT RATING AND RECORDING THE FINAL SCORE. 
This is the next-to-last step in processing the writing examination 



scores. It involves three steps: 



A. Replacing the most discrepant rater's rating with the 



35 



-32- 



referee's rating. There are three possible cases here, one 
where the discrepant rating is immediately apparent - 9 one in 
which it is not, and one special case where there is a borderline 
pass/fail situation. 

1; The discrepant rating is immediately identifiable in 
the case in which one fating differs from any other 
rating by two or more. Instances of this case would 
include the following sets of rating: 1 3 3, 2 4 4, 3 1 
1,422, 134. In each instance the first rating in 
the set is separated from the others by two or more, and 
it would be discarded and replaced by the referee's 
rating. 

2. The discrepant rating is not immediately identifiable 
in two possible distributions of scores: 12 3 and 2 3 4, 
4. The extreme rating in either direction might be 
considered discrepant. In such cases, follow the rule 
£hat the rating furthest from the arithmetical mean of the 
ratings distribution is to be considered discrepant . The 
mean of the ratings 1 through 4 is 2.5. So in the 
distribution i 2 3, the rating of 1 will be discarded and 
replaced by the referee's score, and in the distribution 

2 3 4, the 4 fating will be discarded arid replaced. 

3. The special case is the distribution 112, with two 
faters finding the essay unsatisfactory and one finding 
it satisfactory. Although there is strictly no major 
divergence of judgment here, for the sake of fairness to 
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the testee, all such cases should be sent to a referee. 
Follow this rule for handling the referee's rating in 
this case: 

a. If the referee awards a rating of "2" or higher, 
strike out one of the "1" ratings and replace it 
with the referee's, making the total 1+2+2=5, 
or a passing score. (It would be unlikely for a 
referee to award a "3" in such a case, but if he 
did, the result would be 1+2+3=6.) 

b. If the referee awards a rating of "1", strike 
out the M 2 lf awarded by the rater, which in fact 
transforms the possibly disputed unsatisfactory 
score into a unanimously unsatisfactory one. 

NOTE: It is expected that the function of the referee's 
scoring the essays will be to reduce the variability within sets of 
ratings. Or, to put it differently, that the referee's rating will tend 
to agree with the majority rating or to approach it more closely than 
the discrepant one. In the distribution 1 3 3, for example, it would be 
expected that the referee would award the essay a rating of 2 or 3, thus 
reducing the "spread** of scores. This will, indeed, usually happen, 
since it not likely that the referee happens to share whatever perceptions 
of the essay in question influenced the most discrepant rater. Sometimes, 
however, it will be found that the referee will award the same rating as 
the most discrepant rater—or maybe even a more discrepant one. For the 
sake of the regularity of the procedure, the referee's score in such 
cases still must replace the discrepant score. To do otherwise would be 
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iliegitimateiy to manipulate the reliability estimates for the whole 
operation. 

B. Once the discrepant ratings have been identified and 
stricken out* the score for the essay is totalled by adding the referee's 
rating to the two remaining raters' ratings. This total score is recorded 
in column 6 of the Summary Rater's Tally Sheet. At the same time, as a 
check on accuracy, all other ratings totals should be double-checked and 
also recorded in column 6. 

C. Now, for each essay it is possible to make and record a 
judgment about whether the essay is satisfactory (a passing score) or 
unsatisfactory (a failing score) . 

An.-essay will be deemed satisfactory if the total score 
awa&jed^by^^he three^aters or^ by two raters and a referee is equal 
££i^5— or ^higher . 

An essay will be deemed unsatisfactory if the total score 
awarded by the three raters or two raters and a referee is equal to 
3 or 4 . 

Finally, then, a judgment is to be made about each total score 
recorded in column 6 on the Summary Rater's Tally Sheet— whether it 
passes or fails. In column 7 this judgment is to be recorded by writing 
a "U" if the score is a 3 or a 4 and an "S" if the score is 5 or anything 
higher. 

The Chronology for these operations follows. 

Replacing the Most Discrepant Rater's 
Rating with the Referee's Rating 

i. All the following steps involve only 

28 
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See FIGURE 10, where 
this has been done 



ratings already recorded on the Summary 
copy of each RTS. 

2. For each set of ratings for which there 
is an "X" in column 3 and a referee's 
rating in column 4 : 

a. Locate the most discrepant of the 
three ratings. It would be 

either : 

1. The rating separated by two 
or more ratings from any other 
or 

2. The "I M in the distrib- 
ution 1 2 3, or the "4" in 
the distribution 2 3 4, or 

3* The "1" or "2" rating in a 
112 distribution, where the 
referee's rating is "1" or "2", 
respectively. 

b. Strike through the discrepant 
rating, Clearly marking it as discarded 
but riot obliterating it. 

c. Total the ratings of the. two remaining 
raters and the referee and record the 
score in column 6; 

3. For ail other ratings — i.e., those sets 
not discrepant enough to require 
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that the essays be read by a referee — check 




the arithmetic to assure the proper total 




is recorded and write that total score (from 




column 3) in column 6. 




4. For each score in column .6 that is a 


See FIGURE 11. 


"5" or higher, write an "S" in column 7^ 




signifying that the essay has been judged 




satisfactory. For each score in column 6 




that is a "3" or a "4", write "U" to 




sxgnxry cne essay nas ueen juageu mioauio 




factory. 




6. Repeat these steps for the data on the 




Summary RTS for each set of essays. 







16. FINAL STEPS. With the recording of "S" and "U" grades on the 
Summary RTS's, the ratings process itself is concluded. Disposition 
must be made of the summary RTS 1 s and the sets of essays, however. 

A. One Xerox copy of each Summary RTS is to be made—one copy 
to be filed with the set of essays whose scores it describes; 
the other to be used by a keypunch operator to enter testee 
code numbers and scores into the computer (an operation that 
is beyond the purview of this manual). 
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B. Each cover sheet (removed from the essay in the first 
stages of this process) is to be restapled to the proper 
essay, and the total score awarded the essay (the number 
score, not the "S" or "U" grade) is to be written on each 
cover sheet. When the cover sheets are reattached, each set 
of essays, along with all copies of that set's RTS's, are to 
be placed in the file folder marked for that set, and the 
folders filed in numerical order. 

Reassembling the essays and cover sheets by sets will make 
them easily obtainable and usable in cases where scores are questioned 
or challenged, or where it is desired to use the scored essays for 
research purposes-to perform statistical research on inter-rater reliability, 
for example, or to perform correlational studies on the associations 
between biographical data on the cover sheets and scores on the essays. 

Once the summary RTS's have been gathered up for transmittal 
to the officals responsible for entering the scores into the computer, 
there will be no time pressure on "mopping up 11 the final steps-e.g. , 
reassociating cover sheets with essays-but these final steps should be 
done carefully and accurately so that later access to the record can be 
had without difficulty. 



Transmitting Scores to Testing Office 

1. Assemble all RTS's after ail 

"S" and "U" scores have been determined 
and recorded. 

2. Make one xerox copy of each RTS. 
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3. Assemble one complete set of copies 
of the Summary RTS 1 s completed, during the 
session, place it in a folder or other 
appropriate container, and forward it to 
the designated testing official. 

4. Retain the other set of copies of 
the RTS f s for the purposes of recording 
individual scores on testees cover sheets. 



Reassembling Essays and Cover 
Sheets and Recording Testees Scores 

1. For each set of essays, recover from 
the files the cover sheets removed at 
the beginning of the ratings process. 

2. Staple each cover sheet to its proper 
essay, making sure that the code numbers 
on the two documents match. 

3. When the cover sheets for a set of 
essays have been reattached, refer to 
the Summary RTS for that sat and 
write each testee's total score in 
the space provided for that purpose on 
each cover sheet. 
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4; When ail scores for a set have been 
entered, return the set of essays and 
all copies of the RTS to the folder, to 
be filed for future reference. 

5. Repeat these procedures for all sets 
of essays. 

6. When all sets have been processed 
and filed - 9 place all other papers-- the 
Essay Management Schedules and Referees' 
Schedules — in another marked arid dated 
folder, to be filed with the essays for 
future reference. 
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